Brain activity underlying visual search in depth when viewing volumetric multiplanar images

The study investigated the cortical activity associated with 3D and 2D image perception on a volumetric multiplanar display by analyzing event-related potentials (ERPs) and power spectral density (PSD). In this study, we used a volumetric multiplanar display to present visual targets, and the brain signals were recorded via an EEG amplifier and analyzed using the EEGLAB toolbox on MATLAB. The study found no significant differences in amplitude between the 3D and 2D conditions across five occipital and parietal electrodes. However, there was a significant difference in latency of the P3 component on the Pz electrode. The analysis of PSD showed no significant differences between the two conditions, although there was a slightly higher alpha and beta activity observed in the 2D visualization. The study concluded that 3D image representation on a volumetric multiplanar display has no more sensory or cognitive load on the human brain than 2D representation, and that depth perception on a multiplanar display requires less brain activity.

By developing the three-dimensional (3D) technologies, the demand for high-quality images keeps growing, resulting in the development of various 3D presenter tools in both head-mounted and front-view displays 1,2 . In the field of human-computer interaction, the crucial question is whether the depth effect produced by the new method brings benefits to users to perceive spatial relationships among displayed objects. Answering this question depends on the physical properties of the generated visual stimuli related to the display technology and the specifics of human visual perception 3 . Therefore, assessing the ergonomics of 3D presenter tools has become essential in terms of depth perception and deployment of visual attention 4,5 . Visual search in 3D environment depends on depth perception 6,7 because it relies on our ability to selectively attend to certain features or objects in a visual scene and perceiving them, moreover, the depth cues have an essential role in depth judgment, and applying more depth cues results in more direct depth perception 8,9 . Depth cues are sources of information on the weight of changes depending on the viewing condition 10 . At close viewing distances, the relative binocular disparity is considered a prerequisite for accurate depth perception 10,11 . Visual search in 3D visualization depends on depth perception 6,7,12 because it relies on our ability to selectively attend to certain features or objects in a visual scene and perceiving them, moreover, classical models of visual search 13,14 propose that attention plays an active role in both early and late visual processing. Studies have found that the brain processes information about depth, which is available early in visual processing and becomes more prominent in higher-order representations. Specifically, differences in brain activity between viewing stereoscopic and two-dimensional images have been observed in parietal and occipital regions at 90-130 ms after visual stimulus onset 15 used functional magnetic resonance imaging (fMRI) to examine stereo processing in V1 and other areas of visual cortex 16 , studied evoked potential components that are generated exclusively by cortical structures when dynamic random-dot stereograms (dRDS) were presented 17 , explored cortical responses to dichotically presented random-dot (RD) stimuli which formed a checkerboard by means of horizontal disparity by using fMRI 18 , studied slow cortical potentials and source localization to identify the neural correlates of monocular and binocular depth cues 19 , studied neurophysiological correlates of vertical disparity in 3D images using EEG and specifically P1 component of ERP, and 20 concluded real-world objects trigger stronger and more sustained action-related brain responses than images do. This highlights the important role of these brain areas on processing depth information. Some studies have reported slightly earlier 21 or later 22,23 reactions to depth perception in the brain.
In addition to neural indicators of early sensitivity to depth, highlighted differences in the later cognitive processes 7,[23][24][25] showing that amplitudes correlated with the absolute values of binocular disparities 24  www.nature.com/scientificreports/ orienting of attention 26 at 150-200 ms. In later times, the stereoscopic input modulated high-level perceptual processes involved in the integration of information [24][25][26][27] , figure-ground segmentation 7,23,25 , view generalization 21 , and recognition 19 . These processes were primarily associated with neural activity in the parietal region rather than the occipital region.
Most of the research centered on clarifying the neural activity correlates to the binocular disparity processing that was performed using stereoscopic images ranging from anaglyph-based 21,23,27 to polarization-based 19,28 , however, to the best of our knowledge, there are no studies about the actual depth perception and the direct brain-behavior of actual depth perception. Stereoscopy creates the illusion of image depth by separating visual inputs for both eyes, thus possibly causing binocular vision stress due to the accommodation-vergence conflict that can induce discomfort and visual fatigue 8,[28][29][30][31][32] . On the other hand, actual depth is free of the mentioned issues since no conflict between accommodation and vergence is present. Electroencephalography (EEG) is a reliable method because of its high temporal resolution in order to measure the interaction of human ergonomic properties with physical factors, especially in the field of visual system evaluation for instance 29,31,33 , studied on mental and visual fatigue caused by 3D TV 30 , Quality of experience of stereoscopic 3D TV 28,32 , discomfort evaluation and visual comfort estimation during stereoscopic visualization. Compared to behavioral measures, the neurophysiological one is less biased 28 .
New approaches are being developed to avoid the forceful separation of views for both eyes to improve user comfort and performance. The new displays aim to provide three-dimensional images on multiple focal planes, that it requires no polarized or anaglyph eyewear 1,8,[34][35][36] . Specifically, image points in the physical space of optical elements can be shown in a time-multiplexed manner 1,35,37 . Depth perception plays an essential role in visual search when finding objects in space or information in spatial images. It is crucial both in our daily lives and in performing professional tasks that involve qualitative or quantitative judgments about the third dimension of images. As new three-dimensional systems such as virtual reality, augmented reality, holographic 3D displays, and volumetric multiplanar displays are intended to be used daily, it is crucial to understand how their implementation affects user performance for instance, error rate, response time, efficiency, and satisfaction. Previous behavioral studies provided some experimental support for using volumetric multiplanar display instead of stereoscopic images by reporting more accurate judgments on spatial relationships 47 and faster information recognition time 34 . However, to our knowledge, no studies have yet assessed how neural activity changes in response to new 3D displays.
The present exploratory study aimed to assess the EEG features of an actual 3D visual search. We hypothesized that it could cause lower amplitude in ERPs and Power Spectral Densities (PSD) to find the closest visual element when presented in the form of 3D volumetric images compared to the state that no depth difference exists between elements. So far, EEG was broadly employed to study visual search 25,26 and depth perception 15,18,23,38 for stereoscopic visualization, however, the application of these findings is limited for the understanding of information processing for viewing 3D volumetric images. In the current EEG study, an experiment was performed to investigate the electrophysiological indicators of differences depending on the 3D and 2D volumetric multiplanar image as a real 3D perception.

Materials and methods
In this study, we investigated brain activity during a visual search when individuals viewed volumetric images and answered a visual question. Two different image presentations, i.e., 3D and 2D visual stimuli, were applied to compare information processing requirements. The study was approved by the Ethics Committee of the University of Latvia (ZD2019/20807) and was conducted following the Declaration of Helsinki.
Participants. Twenty participants (9 males, 11 females) with a mean age of 25 ± 6 years voluntarily participated in the study. We assessed the optometric visual tests before the experiment to ensure their binocular vision function. The participants' inclusion criteria were the following: normal or corrected-to-normal visual acuity (1.0 or better, decimal units), no binocular dysfunctions, stereoscopic acuity of 40 arcsec or better (assessed using a Titmus stereo test, Stereo Optical Co., Chicago, IL). At the end four participants excluded from the experiment. Two of them are due to failing the optometric criteria and the rest due to noisy brain signals and high rate of mistake in task performance. The remained participants were 16 subjects (6 males, 10 females) with a mean age of 25 ± 6 years. All participants were unaware of the specific purpose of the study.
Apparatus. The solid-state volumetric display (LightSpace Technologies, model: X1406, 19" diagonal) was used to show the visual stimuli. The display contains twenty physical image depth planes and short liquid-crystal-based light diffuser elements acting as temporary image receiving screens when synchronously coupled to a high refresh-rate image projection unit 35 . In operation, diffusers are switched between the highly transparent and light diffusing state at a high rate, ensuring an overall volumetric image refresh rate of 60 Hz. The resolution of the displays is 1024 × 768 pixels per image depth plane.
Procedure and Task. First, the procedure and the aims of the task were explained to the participants. Then, the participants provided written informed consent. Participants' visual acuity and stereoscopic acuity were screened. The experiment was conducted in a scotopic (0.1 lx) condition.
The whole experiment included a total of 160 trials. The 3D demonstration occurred in 50% of the trials in a pseudo-randomized order. Each trial started with a fixation cross that was presented in the middle of the screen for 1 s. Next, four circles (outer diameter -0.5°, line width -0.1°) were displayed at 1.0° field eccentricity from the display center. In the 3D trials, one circle (a target) had a different binocular disparity in comparison to three others (distractors), because of which it appeared closer to the viewer, shown in Figs www.nature.com/scientificreports/ search for a target and report its relative location within the display by choosing one of four responses (up, right, down, and left). For the response, the arrow keys of a computer keyboard were used. After the response submission, the fixation cross reappeared, and the subsequent trial followed. The entire time of the task performance was on average 10 min depending on the response time. All target positions were counterbalanced across directions.
In the 2D trials, all circles were presented at one depth plane; thus, the search elements contained equal disparity. The subjects responded to the 2D trials by pressing the space button on the keyboard. The experiment was not time constrained. However, the participants were instructed to complete the task as accurately and quickly as possible. Participants sat facing the display at a viewing distance of 90 cm (see Fig. 2).
EEG data acquisition and analysis. The brain's electrical activity was captured with a Nicolet v32 EEG system. Twenty-one active electrodes were positioned following the international 10-20 system 39 , and the average of all active electrodes was chosen as a reference. A sampling rate of 1024 Hz was selected for data collection, and a band-pass filter was applied from 1 to 70 Hz. Data were recorded continuously during the visual search tasks whilst electrode impedance was kept below 10 KΩ. Data were saved alongside triggers marking significant events. EEG data analysis was performed using the open-source toolbox EEGLAB 2019.1.0 connected to MAT-LAB R2018a (MathWorks Inc., Natick, MA, USA). The EEG recordings for every participant were purified by the noise rejection, for example, removing baseline, blinking, and muscle activity based on a two-step procedure that included the built-in software algorithm and visual inspection of variance. The signals were separated into time-locked epochs of 1200 ms duration synchronized with the onset of search arrays, containing 200 ms of prestimulus. For everyone, the epochs were averaged according to the stimulus condition (3D and 2D trials). Only correct responses were analyzed, excluding trials with false alarms or misses. Three separate time windows were selected corresponding to different cognitive processes in visual search and depth perception 18,19,24,25,40 to study the EEG signal variation at the temporal level. Thus, the amplitudes were calculated as the mean potentials in the following time windows: 50-100 ms, 100-200 ms, and 200-450 ms. Correspond to components N1, P2, and P3, respectively. The amplitudes of ERPs on five electrodes positioned over the parietal and occipital regions (O1, O2, P3, P4, and Pz) were investigated in detail. Moreover, the Power Spectral Density (PSD) of alpha and beta waves were analyzed to assess the cognitive properties of the volumetric multiplanar image depth perception.
Statistical data analysis. At first, to analyze the ERP data, the given time interval was divided into three subintervals: (1) 50-100 ms; (2) 100-200 ms; (3) 200-450 ms. Then the smallest response value for each subject was determined within the first interval, and the most considerable value was determined within the second and third intervals. The distributions of the obtained values were relatively symmetric with no apparent outliers; hence the paired t-test was used to test for significant differences between the two conditions (for each electrode  www.nature.com/scientificreports/ and subinterval). In total, 15 tests were performed. Hence some adjustment methods such as Bonferroni or Holm must be applied when the results are interpreted. However, in the paper, original p-values are reported.
In the analysis of frequency band data, the paired Wilcoxon signed-rank test was used to test for significant differences between the two conditions at each of the given frequencies for each electrode. Although for most of the frequency, normality could not be rejected, and paired t-tests could be used, which is known to have a more extensive power in detecting significant effects.

Results
Performance Data. The behavioral results indicated that across all participants, the mean correct response rates for 3D stimuli were 0.98 (SD = 0.04); however, more errors were made when individuals responded to 2D stimuli on the volumetric display. Specifically, the correct response rates dropped to 0.81 (SD = 0.06) when the 2D stimuli were shown. In addition to the correct response rates, response times (RT) were analyzed to evaluate visual search outcomes. The statistical analysis revealed in Fig. 3 that there were no significant main effects of the stimulus condition.

Event-related potentials (ERPs).
We analyzed the cortical signals in three time windows corresponding to ERPs' N1, P2, and P3 components. Generally, higher activation of occipital and parietal was seen as expected. Figure 4 shows changes in amplitudes of ERPs at three time windows averaged over all sixteen participants in the form of topographical maps. Moreover, Fig. 5 shows the same time windows in the form of bar chart including information about the standard deviation and p-value.
Waveforms. P1 component is not easily detectable; however, it occurs approximately after 50 ms of stimuli onset 41 . Waveform showed the P1 component was presented as a negative deflection, therefore we had N1  www.nature.com/scientificreports/ component. Since the ERPs components are dominant waves on the occipital and parietal areas, the statistical analysis applied on five electrodes (O1, O2, P3, P4, and Pz). Figure 6 indicates the properties of the waveform results. Since we had N1 component, we considered the minimum point of each waveform to perform the statistical analysis. Moreover, we chose the max value of P2 and P3 components. The results showed there was no significant differences between two conditions in three time-windows across 16 subjects and for each electrode location.
The latency analysis of ERP components showed no statistically significant differences between the two conditions except for the Pz electrode and within the time-window 200-450 ms, which corresponds to the P3 component. The average latency results are summarized and reported in Table 1.

Power spectral density (PSD).
Continuous wave analysis showed slightly higher activation in alpha and beta frequency bands; however, the difference was not statistically significant. Topographical maps and waveforms of alpha and beta showed in Fig. 7a and b, respectively. Ethical approval. The study was approved by the Ethics Committee of the University of Latvia (ZD2019/20,807) and was conducted following the Declaration of Helsinki.  www.nature.com/scientificreports/

Discussion
In this study, we analyzed cortical activity in the human brain by analyzing ERPs and PSD over occipital and parietal electrodes under two conditions, 2D and 3D perception of volumetric visual stimuli representation.
The goal was to determine if there were differences in amplitude and latency of ERP components as well as the amplitude of PSD regarding alpha and beta bands under two viewing conditions. Since this study focused on volumetric display, the results could indicate positive aspects of the interaction of the volumetric display with the human visual system, as there is no accommodation-convergence conflict in this examination. Behavioral data is a logical way to prove or reject a hypothesis. Previous studies show that response time can be used as a factor for visual target perception 11,48 . In this study, behavioral data, which indicated shorter response time in the 3D task, could serve as logical proof of easier perception of 3D images on the volumetric  www.nature.com/scientificreports/ display compared to 2D images, as reported in 34 . Since this study was time-free, the subjects had sufficient time and motivation to perform the task correctly. The effect of the third dimension of the image was already observed in the brain activity associated with sensory resources and early information processing in the occipital and parietal regions at 50-100 ms after stimulus onset. However, its size was small for both visual targets, possibly due to similar physical properties and dimensions of 3D and 2D visual stimuli. Moreover, a slight difference could reflect involvement of early neural activity by receiving feedback from higher-level processing locations [16][17][18][19][20][21][22][23]27 . Interestingly, early perceptual sensitivity of ERPs to depth was reported not only for digital images [16, 19. 21-23, 27] but also for actual objects 20 . Furthermore, early cognitive responses did not vary depending on the type of visual target 27 and the availability of consistent depth cues 28 . These findings suggest that early information processing can be invariant in response to within-dimension features.
The second time window (100-200 ms) is associated with cognitive processes such as working memory, memory performance, and semantic processing 42,43 . Although there was no statistically significant difference between the two types of images, when looking at topographical maps and waveforms, slightly higher activity can be seen in 2D image perception compared to 3D. This could be because presentation of both visual targets in a sequence of trials caused involvement of memory to differentiate the 2D visual target from the 3D one. Moreover, the P2 peak latency result showed no statistical differences between two conditions over five occipital and parietal electrodes across all subjects. This similar latency represents equal attention and memory function involved in visual data analysis by the visual cortex.
The P3 component is a dominant wave on the parietal cortical area that is typically observed after 250 ms of stimuli onset and is associated with decision making. The latency of this wave is influenced by the difficulty of decision making [44][45][46] . In our study, we observed a slightly higher amplitude of P3 associated with 2D perception, although this difference was not statistically significant. However, there was a significant difference in P3 latency between the 2D and 3D conditions over the Pz electrode. This finding suggests that discriminating 2D stimuli on a volumetric display may be more challenging for subjects performing 3D tasks. Interestingly, our results contradict those of 28 , who reported higher P3 amplitude in the comfort zone of virtual depth perception compared to the non-comfort zone.
Power Spectral Density (PSD) is a measure of the distribution of signal power over frequency, which can provide insight into how the brain reacts to stimuli. For example, higher activity in the alpha band is typically associated with lower attention, while higher activity in the beta band indicates a higher cognitive or problemsolving load on the brain 2 . Our analysis of PSD revealed that alpha and beta had slightly higher amplitudes (though not significant) during the 2D task, which may be due to additional processing demands required to discriminate 3D from 2D trials in the 2D experiment. Additionally, we examined parameters such as total power, spectral band power, and median and spectral edge frequency, which can provide further information about brain activity in response to stimuli.

Conclusion
In this experiment, we studied the effect of the 2D and the 3D volumetric multiplanar image on human brain signals by analyzing EEG data. Event-related potentials in three time windows corresponding to N1, P2, and P3 components were analyzed, as well as the power spectral density of two frequency bands. In this work, generally, no significant differences over three time windows and alpha and beta frequency bands in two conditions were in line with our hypothesis that 3D perception on the volumetric multiplanar display has no extra load on human brain processing.

Data availability
The analyzed data of ERP and frequency bands are available in google drive, https:// drive. google. com/ drive/ folde rs/ 1ZjsX rSywL-1LCm6 VC2YO U71KB-twfJhf. The statistic analysis is available from the corresponding author on reasonable request.